Autonomous vehicle remote support mapping interface

ABSTRACT

Methods and systems for remote support of autonomous operation of vehicles have been disclosed. State indicators are generated by a first state display based on state data from a portion of vehicles assigned to a respective first level control station. A second state display is generated for a second control station and displays state indicators for the state data of the vehicles. A remote support interface including the first state display and image data received from a first vehicle of the vehicles is generated. Instruction data to the first vehicle is transmitted using the remote support interface and based on an indication that the first vehicle needs remote support, the instruction data modifying the autonomous operation of the first vehicle. A workload between the first level control stations is allocated by assigning the vehicles using the state indicators of the second state display.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application claims priority to and the benefit of U.S. ProvisionalApplication Ser. No. 62/428,026, filed Nov. 30, 2016, the entiredisclosure of which is hereby incorporated by reference.

TECHNICAL FIELD

This application relates to an autonomous vehicle mapping interface,including methods, apparatuses, systems, and non-transitory computerreadable media for remote support and tele-operation of vehicles,including autonomous vehicles, using the mapping interface.

BACKGROUND

Traditional traffic control management is centered on dealing with aninfrastructure that is only partly subject to control by humanoperators. For example, human driven vehicles are mostly beyond thereach of operators except in the form of communication with the driver(e.g. providing road-side assistance instructions). Autonomous vehiclesenable operators to remotely control and support the autonomousvehicles.

SUMMARY

Disclosed herein are aspects, features, elements, implementations, andimplementations for remote support of vehicles.

In an implementation, a remote support apparatus is provided. The remotesupport apparatus comprises a memory and a processor configured toexecute instructions stored in the memory to: receive state data fromvehicles, generate a first map display including a representation of ageographical area and vehicle map indicators representing a portion ofthe vehicles within the geographical area, generate a first statedisplay including a representation of the portion of the vehicles asstate indicators, generate a remote support interface including thefirst state display and image data received from a first vehicle of theportion of vehicles, wherein the remote support interface is responsiveto an input signal to a first state indicator of the first vehicle fromthe first state display; and transmit instruction data to the firstvehicle to modify an autonomous operation of the first vehicle based onthe input signal to the remote support interface.

In an implementation, a remote support apparatus is provided. The remotesupport apparatus comprises a memory and a processor configured toexecute instructions stored in the memory to: receive state data fromvehicles, rank the vehicles based on a level of urgency for remotesupport of the vehicles, the level of urgency determined using the statedata, generate a first state display that displays the state data of atleast some of the vehicles as respective state indicators arranged basedon the level of urgency, generate a remote support interface includingthe first state display and image data received from a first vehicle ofthe at least some of the vehicles, the remote support interfaceresponsive to an input signal to a first state indicator of the firstvehicle from the first state display, and transmit a signal to the firstvehicle using the remote support interface.

In an implementation, a method for remote support of autonomousoperation of vehicles is provided. The method comprises receiving statedata from the vehicles, generating, for first level control stations, arespective first state display that displays the state data from aportion of the vehicles assigned to a respective one of the first levelcontrol stations as respective state indicators, generating, for asecond level control station, a second state display that displays thestate data of the vehicles, generating a remote support interfaceincluding the first state display and image data received from a firstvehicle of the vehicles; transmitting, using the remote supportinterface, instruction data to the first vehicle based on an indicationthat the first vehicle needs remote support, the instruction data, oncereceived by the first vehicle, modifying autonomous operation of thefirst vehicle, and allocating a workload between the first level controlstations by assigning the vehicles using the state indicators of thesecond state display.

These and other aspects of the present disclosure are disclosed in thefollowing detailed description of the embodiments, the appended claimsand the accompanying figures.

BRIEF DESCRIPTION OF THE DRAWINGS

The disclosed technology is best understood from the following detaileddescription when read in conjunction with the accompanying drawings. Itis emphasized that, according to common practice, the various featuresof the drawings are not to-scale. On the contrary, the dimensions of thevarious features are arbitrarily expanded or reduced for clarity.

FIG. 1 is a diagram of an example of a portion of a vehicle in which theaspects, features, and elements disclosed herein may be implemented.

FIG. 2 is a diagram of an example of a portion of a vehicletransportation and communication system in which the aspects, features,and elements disclosed herein may be implemented.

FIG. 3 is a block diagram illustrating a remote vehicle assistancecenter in accordance with the present disclosure.

FIG. 4 is a diagram illustrating an example of a vehicle indicator foruse in an interface in accordance with the present disclosure.

FIG. 5 is a diagram illustrating an example of a fleet manager interfacein accordance with the present disclosure.

FIG. 6 is a diagram illustrating an example of a vehicle indicator foruse in an interface in accordance with the present disclosure.

FIG. 7 is a diagram illustrating an example of a vehicle managerinterface in accordance with the present disclosure.

FIG. 8 is a diagram illustrating an example of a vehicle managerinterface in accordance with the present disclosure.

FIG. 9 is a diagram illustrating an example of a vehicle managerinterface in accordance with the present disclosure.

FIG. 10 is a diagram illustrating an example of a vehicle managerinterface in accordance with the present disclosure.

FIG. 11 is a flow chart of a technique for providing remote support ofautonomous operation of vehicles in accordance with the presentdisclosure.

FIG. 12 is a flow chart of a technique for providing remote support ofautonomous operation of vehicles in accordance with the presentdisclosure.

FIG. 13 is a flow chart of a technique for providing remote support ofautonomous operation of vehicles in accordance with the presentdisclosure.

FIG. 14 illustrates a method for remote support in accordance with thepresent disclosure.

DETAILED DESCRIPTION

Managing large numbers of vehicles, including autonomous vehicles, oftenresults in the creation and processing of a correspondingly large amountof data. Much of this data ends up being viewed and used by the humanoperators tasked with management of the vehicles. As such, the way inwhich the data is organized and distributed to operators can affect theeffectiveness with which that the operators use the data to manage andprovide remote support to the vehicles.

The disclosed technology provides a more effective interface foroperator interaction with vehicles, including an enhanced way ofgenerating indicators on a remote operator interface, so that thevehicles can be arranged according to their priority level (e.g. theurgency with which assistance is requested by a vehicle) or otheraggregated data and factors. In this way, the operators tasked withmanaging the vehicles can efficiently view and interact with relevantdata. The disclosed technology offers a streamlined way for an operatorto assist a vehicle, update the vehicle's assistance state (e.g. issueresolved), and move on to the next vehicle that requires assistance. Theinformation (including status or state information) of the vehicle canbe aggregated into various indicators that display pertinent informationrelated to the vehicles (e.g., travel time remaining, number ofpassengers, health of the car). The indicators can then be assigned tovarious control stations managed by vehicle managers. The assignment canbe done manually by a fleet manager or can be done automatically usingmachine learning techniques or aggregated historical information.

As used herein, the terminology “driver” or “operator” may be usedinterchangeably. As used herein, the terminology “brake” or “decelerate”may be used interchangeably. As used herein, the terminology “computer”or “computing device” includes any unit, or combination of units,capable of performing any method, or any portion or portions thereof,disclosed herein.

As used herein, the terminology “processor” indicates one or moreprocessors, such as one or more special purpose processors, one or moredigital signal processors, one or more microprocessors, one or morecontrollers, one or more microcontrollers, one or more applicationprocessors, one or more Application Specific Integrated Circuits, one ormore Application Specific Standard Products; one or more FieldProgrammable Gate Arrays, any other type or combination of integratedcircuits, one or more state machines, or any combination thereof.

As used herein, the terminology “memory” indicates any computer-usableor computer-readable medium or device that can tangibly contain, store,communicate, or transport any signal or information that may be used byor in connection with any processor. For example, a memory may be one ormore read only memories (ROM), one or more random access memories (RAM),one or more registers, low power double data rate (LPDDR) memories, oneor more cache memories, one or more semiconductor memory devices, one ormore magnetic media, one or more optical media, one or moremagneto-optical media, or any combination thereof.

As used herein, the terminology “instructions” may include directions orexpressions for performing any method, or any portion or portionsthereof, disclosed herein, and may be realized in hardware, software, orany combination thereof. For example, instructions may be implemented asinformation, such as a computer program, stored in memory that may beexecuted by a processor to perform any of the respective methods,algorithms, aspects, or combinations thereof, as described herein. Insome implementations, instructions, or a portion thereof, may beimplemented as a special purpose processor, or circuitry, that mayinclude specialized hardware for carrying out any of the methods,algorithms, aspects, or combinations thereof, as described herein. Insome implementations, portions of the instructions may be distributedacross multiple processors on a single device, on multiple devices,which may communicate directly or across a network such as a local areanetwork, a wide area network, the Internet, or a combination thereof.

As used herein, the terminology “example,” “embodiment,”“implementation,” “aspect,” “feature,” or “element” indicate serving asan example, instance, or illustration. Unless expressly indicated, anyexample, embodiment, implementation, aspect, feature, or element isindependent of each other example, embodiment, implementation, aspect,feature, or element and may be used in combination with any otherexample, embodiment, implementation, aspect, feature, or element.

As used herein, the terminology “determine” and “identify,” or anyvariations thereof, includes selecting, ascertaining, computing, lookingup, receiving, determining, establishing, obtaining, or otherwiseidentifying or determining in any manner whatsoever using one or more ofthe devices shown and described herein.

As used herein, the terminology “or” is intended to mean an inclusive“or” rather than an exclusive “or.” That is, unless specified otherwise,or clear from context, “X includes A or B” is intended to indicate anyof the natural inclusive permutations. If X includes A; X includes B; orX includes both A and B, then “X includes A or B” is satisfied under anyof the foregoing instances. In addition, the articles “a” and “an” asused in this application and the appended claims should generally beconstrued to mean “one or more” unless specified otherwise or clear fromcontext to be directed to a singular form.

Further, for simplicity of explanation, although the figures anddescriptions herein may include sequences or series of steps or stages,elements of the methods disclosed herein may occur in various orders orconcurrently. Additionally, elements of the methods disclosed herein mayoccur with other elements not explicitly presented and described herein.Furthermore, not all elements of the methods described herein may berequired to implement a method in accordance with this disclosure.Although aspects, features, and elements are described herein inparticular combinations, each aspect, feature, or element may be usedindependently or in various combinations with or without other aspects,features, and elements.

U.S. patent application Ser. No. 15/463,242, filed on Mar. 20, 2017,entitled “OBJECT MANAGEMENT DISPLAY,” is incorporated herein byreference in its entirety.

To describe some implementations in greater detail, reference is made tothe following figures.

FIG. 1 is a diagram of an example of a vehicle 1000 in which theaspects, features, and elements disclosed herein may be implemented. Thevehicle 1000 includes a chassis 1100, a powertrain 1200, a controller1300, wheels 1400/1410/1420/1430, or any other element or combination ofelements of a vehicle. Although the vehicle 1000 is shown as includingfour wheels 1400/1410/1420/1430 for simplicity, any other propulsiondevice or devices, such as a propeller or tread, may be used. In FIG. 1,the lines interconnecting elements, such as the powertrain 1200, thecontroller 1300, and the wheels 1400/1410/1420/1430, indicate thatinformation, such as data or control signals, power, such as electricalpower or torque, or both information and power, may be communicatedbetween the respective elements. For example, the controller 1300 mayreceive power from the powertrain 1200 and communicate with thepowertrain 1200, the wheels 1400/1410/1420/1430, or both, to control thevehicle 1000, which can include accelerating, decelerating, steering, orotherwise controlling the vehicle 1000.

The powertrain 1200 includes a power source 1210, a transmission 1220, asteering unit 1230, a vehicle actuator 1240, or any other element orcombination of elements of a powertrain, such as a suspension, a driveshaft, axles, or an exhaust system. Although shown separately, thewheels 1400/1410/1420/1430 may be included in the powertrain 1200.

The power source 1210 may be any device or combination of devicesoperative to provide energy, such as electrical energy, thermal energy,or kinetic energy. For example, the power source 1210 includes anengine, such as an internal combustion engine, an electric motor, or acombination of an internal combustion engine and an electric motor, andis operative to provide kinetic energy as a motive force to one or moreof the wheels 1400/1410/1420/1430. In some embodiments, the power source1210 includes a potential energy unit, such as one or more dry cellbatteries, such as nickel-cadmium (NiCd), nickel-zinc (NiZn), nickelmetal hydride (NiMH), lithium-ion (Li-ion); solar cells; fuel cells; orany other device capable of providing energy.

The transmission 1220 receives energy, such as kinetic energy, from thepower source 1210, and transmits the energy to the wheels1400/1410/1420/1430 to provide a motive force. The transmission 1220 maybe controlled by the controller 1300, the vehicle actuator 1240 or both.The steering unit 1230 may be controlled by the controller 1300, thevehicle actuator 1240, or both and controls the wheels1400/1410/1420/1430 to steer the vehicle. The vehicle actuator 1240 mayreceive signals from the controller 1300 and may actuate or control thepower source 1210, the transmission 1220, the steering unit 1230, or anycombination thereof to operate the vehicle 1000.

In some embodiments, the controller 1300 includes a location unit 1310,an electronic communication unit 1320, a processor 1330, a memory 1340,a user interface 1350, a sensor 1360, an electronic communicationinterface 1370, or any combination thereof. Although shown as a singleunit, any one or more elements of the controller 1300 may be integratedinto any number of separate physical units. For example, the userinterface 1350 and processor 1330 may be integrated in a first physicalunit and the memory 1340 may be integrated in a second physical unit.Although not shown in FIG. 1, the controller 1300 may include a powersource, such as a battery. Although shown as separate elements, thelocation unit 1310, the electronic communication unit 1320, theprocessor 1330, the memory 1340, the user interface 1350, the sensor1360, the electronic communication interface 1370, or any combinationthereof can be integrated in one or more electronic units, circuits, orchips.

In some embodiments, the processor 1330 includes any device orcombination of devices capable of manipulating or processing a signal orother information now-existing or hereafter developed, including opticalprocessors, quantum processors, molecular processors, or a combinationthereof. For example, the processor 1330 may include one or more specialpurpose processors, one or more digital signal processors, one or moremicroprocessors, one or more controllers, one or more microcontrollers,one or more integrated circuits, one or more an Application SpecificIntegrated Circuits, one or more Field Programmable Gate Array, one ormore programmable logic arrays, one or more programmable logiccontrollers, one or more state machines, or any combination thereof. Theprocessor 1330 may be operatively coupled with the location unit 1310,the memory 1340, the electronic communication interface 1370, theelectronic communication unit 1320, the user interface 1350, the sensor1360, the powertrain 1200, or any combination thereof. For example, theprocessor may be operatively coupled with the memory 1340 via acommunication bus 1380.

In some embodiments, the processor 1330 may be configured to executeinstructions including instructions for remote operation which may beused to operate the vehicle 1000 from a remote location including theoperations center. The instructions for remote operation may be storedin the vehicle 1000 or received from an external source such as atraffic management center, or server computing devices, which mayinclude cloud based server computing devices.

The memory 1340 may include any tangible non-transitory computer-usableor computer-readable medium, capable of, for example, containing,storing, communicating, or transporting machine readable instructions orany information associated therewith, for use by or in connection withthe processor 1330. The memory 1340 is, for example, one or more solidstate drives, one or more memory cards, one or more removable media, oneor more read only memories, one or more random access memories, one ormore disks, including a hard disk, a floppy disk, an optical disk, amagnetic or optical card, or any type of non-transitory media suitablefor storing electronic information, or any combination thereof.

The electronic communication interface 1370 may be a wireless antenna,as shown, a wired communication port, an optical communication port, orany other wired or wireless unit capable of interfacing with a wired orwireless electronic communication medium 1500.

The electronic communication unit 1320 may be configured to transmit orreceive signals via the wired or wireless electronic communicationmedium 1500, such as via the electronic communication interface 1370.Although not explicitly shown in FIG. 1, the electronic communicationunit 1320 is configured to transmit, receive, or both via any wired orwireless communication medium, such as radio frequency (RF), ultraviolet (UV), visible light, fiber optic, wire line, or a combinationthereof. Although FIG. 1 shows a single one of the electroniccommunication unit 1320 and a single one of the electronic communicationinterface 1370, any number of communication units and any number ofcommunication interfaces may be used. In some embodiments, theelectronic communication unit 1320 can include a dedicated short rangecommunications (DSRC) unit, a wireless safety unit (WSU), IEEE 802.11p(Wifi-P), or a combination thereof.

The location unit 1310 may determine geolocation information, includingbut not limited to longitude, latitude, elevation, direction of travel,or speed, of the vehicle 1000. For example, the location unit includes aglobal positioning system (GPS) unit, such as a Wide Area AugmentationSystem (WAAS) enabled National Marine-Electronics Association (NMEA)unit, a radio triangulation unit, or a combination thereof. The locationunit 1310 can be used to obtain information that represents, forexample, a current heading of the vehicle 1000, a current position ofthe vehicle 1000 in two or three dimensions, a current angularorientation of the vehicle 1000, or a combination thereof.

The user interface 1350 may include any unit capable of being used as aninterface by a person, including any of a virtual keypad, a physicalkeypad, a touchpad, a display, a touchscreen, a speaker, a microphone, avideo camera, a sensor, and a printer. The user interface 1350 may beoperatively coupled with the processor 1330, as shown, or with any otherelement of the controller 1300. Although shown as a single unit, theuser interface 1350 can include one or more physical units. For example,the user interface 1350 includes an audio interface for performing audiocommunication with a person, and a touch display for performing visualand touch based communication with the person.

The sensor 1360 may include one or more sensors, such as an array ofsensors, which may be operable to provide information that may be usedto control the vehicle. The sensor 1360 can provide informationregarding current operating characteristics of the vehicle or itssurrounding. The sensors 1360 include, for example, a speed sensor,acceleration sensors, a steering angle sensor, traction-related sensors,braking-related sensors, or any sensor, or combination of sensors, thatis operable to report information regarding some aspect of the currentdynamic situation of the vehicle 1000.

In some embodiments, the sensor 1360 may include sensors that areoperable to obtain information regarding the physical environmentsurrounding the vehicle 1000. For example, one or more sensors detectroad geometry and obstacles, such as fixed obstacles, vehicles,cyclists, and pedestrians. In some embodiments, the sensor 1360 can beor include one or more video cameras, laser-sensing systems,infrared-sensing systems, acoustic-sensing systems, or any othersuitable type of on-vehicle environmental sensing device, or combinationof devices, now known or later developed. In some embodiments, thesensor 1360 and the location unit 1310 are combined.

Although not shown separately, the vehicle 1000 may include a trajectorycontroller. For example, the controller 1300 may include a trajectorycontroller. The trajectory controller may be operable to obtaininformation describing a current state of the vehicle 1000 and a routeplanned for the vehicle 1000, and, based on this information, todetermine and optimize a trajectory for the vehicle 1000. In someembodiments, the trajectory controller outputs signals operable tocontrol the vehicle 1000 such that the vehicle 1000 follows thetrajectory that is determined by the trajectory controller. For example,the output of the trajectory controller can be an optimized trajectorythat may be supplied to the powertrain 1200, the wheels1400/1410/1420/1430, or both. In some embodiments, the optimizedtrajectory can be control inputs such as a set of steering angles, witheach steering angle corresponding to a point in time or a position. Insome embodiments, the optimized trajectory can be one or more paths,lines, curves, or a combination thereof.

One or more of the wheels 1400/1410/1420/1430 may be a steered wheel,which is pivoted to a steering angle under control of the steering unit1230, a propelled wheel, which is torqued to propel the vehicle 1000under control of the transmission 1220, or a steered and propelled wheelthat steers and propels the vehicle 1000.

A vehicle may include units, or elements not shown in FIG. 1, such as anenclosure, a Bluetooth® module, a frequency modulated (FM) radio unit, aNear Field Communication (NFC) module, a liquid crystal display (LCD)display unit, an organic light-emitting diode (OLED) display unit, aspeaker, or any combination thereof.

FIG. 2 is a diagram of an example of a portion of a vehicletransportation and communication system 2000 in which the aspects,features, and elements disclosed herein may be implemented. The vehicletransportation and communication system 2000 includes a vehicle 2100,such as the vehicle 1000 shown in FIG. 1, and one or more externalobjects, such as an external object 2110, which can include any form oftransportation, such as the vehicle 1000 shown in FIG. 1, a pedestrian,cyclist, as well as any form of a structure, such as a building. Thevehicle 2100 may travel via one or more portions of a transportationnetwork 2200, and may communicate with the external object 2110 via oneor more of an electronic communication network 2300. Although notexplicitly shown in FIG. 2, a vehicle may traverse an area that is notexpressly or completely included in a transportation network, such as anoff-road area. In some embodiments the transportation network 2200 mayinclude one or more of a vehicle detection sensor 2202, such as aninductive loop sensor, which may be used to detect the movement ofvehicles on the transportation network 2200.

The electronic communication network 2300 may be a multiple accesssystem that provides for communication, such as voice communication,data communication, video communication, messaging communication, or acombination thereof, between the vehicle 2100, the external object 2110,and an operations center 2400. For example, the vehicle 2100 or theexternal object 2110 may receive information, such as informationrepresenting the transportation network 2200, from the operations center2400 via the electronic communication network 2300.

The operations center 2400 includes a controller apparatus 2410 whichincludes some or all of the features of the controller 1300 shown inFIG. 1. The controller apparatus 2410 can monitor and coordinate themovement of vehicles, including autonomous vehicles. The controllerapparatus 2410 may monitor the state or condition of vehicles, such asthe vehicle 2100, and external objects, such as the external object2110. The controller apparatus 2410 can receive vehicle data andinfrastructure data including any of: vehicle velocity; vehiclelocation; vehicle operational state; vehicle destination; vehicle route;vehicle sensor data; external object velocity; external object location;external object operational state; external object destination; externalobject route; and external object sensor data.

Further, the controller apparatus 2410 can establish remote control overone or more vehicles, such as the vehicle 2100, or external objects,such as the external object 2110. In this way, the controller apparatus2410 may tele-operate the vehicles or external objects from a remotelocation. The controller apparatus 2410 may exchange (send or receive)state data with vehicles, external objects, or computing devices such asthe vehicle 2100, the external object 2110, or a server computing device2500, via a wireless communication link such as the wirelesscommunication link 2380 or a wired communication link such as the wiredcommunication link 2390.

The server computing device 2500 may include one or more servercomputing devices which may exchange (send or receive) state signal datawith one or more vehicles or computing devices including the vehicle2100, the external object 2110, or the operations center 2400, via theelectronic communication network 2300.

In some embodiments, the vehicle 2100 or the external object 2110communicates via the wired communication link 2390, a wirelesscommunication link 2310/2320/2370, or a combination of any number ortypes of wired or wireless communication links. For example, as shown,the vehicle 2100 or the external object 2110 communicates via aterrestrial wireless communication link 2310, via a non-terrestrialwireless communication link 2320, or via a combination thereof. In someimplementations, a terrestrial wireless communication link 2310 includesan Ethernet link, a serial link, a Bluetooth link, an infrared (IR)link, an ultraviolet (UV) link, or any link capable of providing forelectronic communication.

A vehicle, such as the vehicle 2100, or an external object, such as theexternal object 2110 may communicate with another vehicle, externalobject, or the operations center 2400. For example, a host, or subject,vehicle 2100 may receive one or more automated inter-vehicle messages,such as a basic safety message (BSM), from the operations center 2400,via a direct communication link 2370, or via an electronic communicationnetwork 2300. For example, operations center 2400 may broadcast themessage to host vehicles within a defined broadcast range, such as threehundred meters, or to a defined geographical area. In some embodiments,the vehicle 2100 receives a message via a third party, such as a signalrepeater (not shown) or another remote vehicle (not shown). In someembodiments, the vehicle 2100 or the external object 2110 transmits oneor more automated inter-vehicle messages periodically based on a definedinterval, such as one hundred milliseconds.

Automated inter-vehicle messages may include vehicle identificationinformation, geospatial state information, such as longitude, latitude,or elevation information, geospatial location accuracy information,kinematic state information, such as vehicle acceleration information,yaw rate information, speed information, vehicle heading information,braking system state data, throttle information, steering wheel angleinformation, or vehicle routing information, or vehicle operating stateinformation, such as vehicle size information, headlight stateinformation, turn signal information, wiper state data, transmissioninformation, or any other information, or combination of information,relevant to the transmitting vehicle state. For example, transmissionstate information indicates whether the transmission of the transmittingvehicle is in a neutral state, a parked state, a forward state, or areverse state.

In some embodiments, the vehicle 2100 communicates with the electroniccommunication network 2300 via an access point 2330. The access point2330, which may include a computing device, may be configured tocommunicate with the vehicle 2100, with the electronic communicationnetwork 2300, with the operations center 2400, or with a combinationthereof via wired or wireless communication links 2310/2340. Forexample, an access point 2330 is a base station, a base transceiverstation (BTS), a Node-B, an enhanced Node-B (eNode-B), a Home Node-B(HNode-B), a wireless router, a wired router, a hub, a relay, a switch,or any similar wired or wireless device. Although shown as a singleunit, an access point can include any number of interconnected elements.

The vehicle 2100 may communicate with the electronic communicationnetwork 2300 via a satellite 2350, or other non-terrestrialcommunication device. The satellite 2350, which may include a computingdevice, may be configured to communicate with the vehicle 2100, with theelectronic communication network 2300, with the operations center 2400,or with a combination thereof via one or more communication links2320/2360. Although shown as a single unit, a satellite can include anynumber of interconnected elements.

The electronic communication network 2300 may be any type of networkconfigured to provide for voice, data, or any other type of electroniccommunication. For example, the electronic communication network 2300includes a local area network (LAN), a wide area network (WAN), avirtual private network (VPN), a mobile or cellular telephone network,the Internet, or any other electronic communication system. Theelectronic communication network 2300 may use a communication protocol,such as the transmission control protocol (TCP), the user datagramprotocol (UDP), the internet protocol (IP), the real-time transportprotocol (RTP) the Hyper Text Transport Protocol (HTTP), or acombination thereof. Although shown as a single unit, an electroniccommunication network can include any number of interconnected elements.

In some embodiments, the vehicle 2100 communicates with the operationscenter 2400 via the electronic communication network 2300, access point2330, or satellite 2350. The operations center 2400 may include one ormore computing devices, which are able to exchange (send or receive)data from: vehicles such as the vehicle 2100; external objects includingthe external object 2110; or computing devices such as the servercomputing device 2500.

In some embodiments, the vehicle 2100 identifies a portion or conditionof the transportation network 2200. For example, the vehicle 2100 mayinclude one or more on-vehicle sensors 2102, such as the sensor 1360shown in FIG. 1, which includes a speed sensor, a wheel speed sensor, acamera, a gyroscope, an optical sensor, a laser sensor, a radar sensor,a sonic sensor, or any other sensor or device or combination thereofcapable of determining or identifying a portion or condition of thetransportation network 2200.

The vehicle 2100 may traverse one or more portions of the transportationnetwork 2200 using information communicated via the electroniccommunication network 2300, such as information representing thetransportation network 2200, information identified by one or moreon-vehicle sensors 2102, or a combination thereof. The external object2110 may be capable of all or some of the communications and actionsdescribed above with respect to the vehicle 2100.

For simplicity, FIG. 2 shows the vehicle 2100 as the host vehicle, theexternal object 2110, the transportation network 2200, the electroniccommunication network 2300, and the operations center 2400. However, anynumber of vehicles, networks, or computing devices may be used. In someembodiments, the vehicle transportation and communication system 2000includes devices, units, or elements not shown in FIG. 2. Although thevehicle 2100 or external object 2110 is shown as a single unit, avehicle can include any number of interconnected elements.

Although the vehicle 2100 is shown communicating with the operationscenter 2400 via the electronic communication network 2300, the vehicle2100 (and external object 2110) may communicate with the operationscenter 2400 via any number of direct or indirect communication links.For example, the vehicle 2100 or external object 2110 may communicatewith the operations center 2400 via a direct communication link, such asa Bluetooth communication link. Although, for simplicity, FIG. 2 showsone of the transportation network 2200, and one of the electroniccommunication network 2300, any number of networks or communicationdevices may be used.

FIG. 3 is a block diagram illustrating a remote vehicle assistancecenter 3000 according to this disclosure. The remote vehicle assistancecenter 3000 includes a fleet manager 3010, a plurality of vehiclemanagers including but not limited to a vehicle manager 3020 and avehicle manager 3030, and a plurality of vehicles including but notlimited to vehicles 3040, 3050, 3060, and 3070.

The fleet manager 3010 can include an apparatus including some or all ofthe features of the controller 1300 shown in FIG. 1. The fleet manager3010 can monitor and coordinate vehicle managers, including the vehiclemanagers 3020/3030 as well as the movement of vehicles, includingautonomous vehicles, and the vehicles 3040/3050/3060/3070. Monitoringand coordinating the vehicle managers can include any of: assigning,allocating, or deallocating, vehicles to the vehicle managers; reviewingand monitoring performance data of the vehicle managers; and assigningvehicle managers to a geographic area. In an implementation, there canbe multiple fleet managers, who may in turn be managed or under theauthority of other fleet managers.

The vehicle manager 3020 can monitor the state or condition of vehicles,including the vehicle 3040 and the vehicle 3050. As illustrated in FIG.3, the vehicle manager 3020 has been assigned vehicle 3040 and vehicle3050. The assignment of vehicles to a vehicle manager can be performedby a fleet manager such as the fleet manager 3010.

The vehicle manager 3030 can monitor the state or condition of vehicles,including the vehicle 3060 and the vehicle 3070. As illustrated in FIG.3, the vehicle manager 3030 has been assigned vehicle 3060 and vehicle3070. The assignment of vehicles to a vehicle manager can be performedby a fleet manager such as the fleet manager 3010. The assignment ofvehicles to a vehicle manager can also be automated using machinelearning techniques.

In an implementation, the vehicle managers can cluster or group thevehicles, establish communication with occupants in the vehicle,remotely operate the vehicles, and coordinate the movement of thevehicles through a transportation network or around various obstaclessuch as traffic congestion. The vehicle managers can interact with othervehicle managers to aid in the monitoring and management of thevehicles.

The vehicles including the vehicle 3040/3050/3060/3070 comprise vehiclessuch as the vehicle 2100 as shown in FIG. 2, that are being monitored orcoordinated by the fleet manager 3010. The vehicles can be operatedautonomously or by a human driver and can exchange (send and receive)vehicle data relating to the state or condition of the vehicle and itssurroundings including any of: vehicle velocity; vehicle location;vehicle operational state; vehicle destination; vehicle route; vehiclesensor data; external object velocity; and external object location.

FIG. 4 is a diagram illustrating an example of a vehicle indicator 4000for use in an interface including a fleet manager interface. The vehicleindicator 4000 includes a task state indicator 4010, a vehicle modeindicator 4020, a vehicle occupancy indicator 4030, and a temporal stateindicator 4040.

The task state indicator 4010 can be used to indicate a task that isbeing performed by a vehicle or that is assigned to the vehicleincluding any of: travelling to a pickup destination, includingtravelling to a destination to pick up one or more passengers or cargo;travelling to a drop-off destination, including travelling to adestination to drop off one or more passengers or cargo; travelling to amaintenance destination, including travelling to a destination at whichmaintenance or repairs can be performed on the vehicle; and travellingto a refueling destination, including travelling to a destination torefuel the vehicle including petroleum stations or electrical chargingstations.

The characteristics of the task state indicator 4010, including theshape and color, can correspond to the task that is being performed bythe vehicle. For example, the task state indicator 4010 is shown as asquare in FIG. 4, which can indicate, for example, that the vehicle istravelling to a pick up destination. In an implementation, a circularshape of the task state indicator 4010 can indicate that the vehicle istravelling to a drop-off destination. Different shapes and colors canindicate different tasks being performed by the vehicle.

The vehicle mode indicator 4020 can be used to indicate whether thevehicle is operating in any of: an autonomous mode, including a mode inwhich the vehicle directs itself or the vehicle is being directedremotely by a computing device; a directed mode, including a mode inwhich a human operator is directing the vehicle from a remote location;a manual mode, including a mode in which a human operator is operatingthe vehicle from within the vehicle; and a semi-autonomous mode,including a mode in which the vehicle can switch between the autonomousmode and the manual mode based on the state of the vehicle (e.g.assistive braking can activate when a proximity and accelerationthreshold is exceeded) and a mode in which the vehicle is simultaneouslybeing controlled using autonomous features and human operation. Forexample, the vehicle mode indicator 4020 is shown as a cross in FIG. 4,which can indicate, for example, any of the aforementioned modes.

The vehicle occupancy indicator 4030 can be used to indicate any of:whether the vehicle contains one or more passengers; and the state ofthe passengers in the vehicle. In an implementation, an occupied vehicleis indicated by the vehicle occupancy indicator 4030 being in a filledstate (e.g. the area inside the vehicle occupancy indicator 4030 is thesame color as the border around the task state indicator 4010).

The characteristics of the vehicle occupancy indicator 4030, includingthe color, can be used to indicate an issue associated with the vehicleincluding any of a passenger issue, including a request for assistancefrom a passenger inside the vehicle, a traffic issue, including issuesrelating to traffic congestion, traffic accidents, and construction, adecision issue, including issues relating to a decision that can be madeby a vehicle manager regarding whether to take control over the vehicle,reroute the vehicle, establish communication with the vehicle, orindicate that an action with respect to the vehicle has been completed,a physical issue with the state of the vehicle, including issuesrelating to the operational state of the vehicle (e.g. engine state,fuel state). In an implementation, a default color can be used toindicate that the vehicle is operating in a normal state and that noissues with the vehicle are pending.

The temporal state indicator 4040 can be used to indicate the temporalstate of the vehicle in relation to an expected or predicted temporalstate. In an implementation, the color of the temporal state indicator4040 can indicate whether the vehicle is ahead of a scheduled time orbehind a scheduled time.

The length of the temporal state indicator 4040 can indicate a magnitudeof deviation from the expected or predicted temporal state. The lengthof the temporal state indicator 4040 can be proportional to themagnitude of the deviation from the expected or predicted temporal state(e.g. directly proportional, inversely proportional, exponentiallyproportional, logarithmic proportionality).

In another implementation, the length of the temporal state indicator4040 can be disproportionate with respect to the deviation from theexpected or predicted temporal state (e.g. a one-third length indicatinga less than five-minute deviation, a two-third length indicating adeviation of more than five minutes and less than fifteen minutes, and afull-length indicating a deviation of greater than fifteen minutes).Other characteristics of the temporal state indicator 4040 can be usedto indicate the state of the vehicle, for example, a red color couldindicate that the vehicle is behind a scheduled time and a green colorcould indicate that the vehicle is ahead of a scheduled time.

FIG. 5 is a diagram illustrating an example of a fleet manager interface5000. The fleet manager interface 5000 can be generated based on one ormore instructions that are executable on a computing apparatus,including the controller apparatus 2410 shown in FIG. 2, and which canbe stored in a memory of a computing apparatus, including the controllerapparatus 2410.

For example, the fleet manager interface 5000 can be generated by thecontroller apparatus 2410, based on instructions that are interpreted bya client computing device that accesses the controller apparatus 2410through a computer network. The client computing device can thengenerate a representation of the fleet manager interface 5000 on adisplay device.

The fleet manager interface 5000 includes a fleet manager portion 5010,a map portion 5020, a vehicle manager indicator 5030, a vehicleindicator 5040, and a vehicle manager assignment queue 5050, any ofwhich can be based on data associated with the state of physical objectsincluding but not limited to at least one of vehicles, roadways,buildings, and pedestrians.

The fleet manager portion 5010 includes a representation of objects thatare being monitored or tracked by the fleet manager and/or the vehiclemanagers including the association of vehicle managers to vehicles. Theobjects can include vehicles, including the vehicle 2100 as shown inFIG. 2. The objects can be represented as indicators such as the vehicleindicator 5040, which can be generated as a static image or moving imagesuch as the vehicle indicator 4000 as shown in FIG. 4. Further, thefleet manager portion 5010 can receive input including any of touchinputs, voice inputs, and inputs from an input device. By way ofexample, vehicle indicators including the vehicle indicator 5040 can beselected by an operator, such as a vehicle manager. Selection of thevehicle indicators can generate data about the state or condition of therespective vehicle represented by the vehicle indicators (e.g. theselected vehicle indicator can indicate whether the vehicle isfunctioning properly or will arrive at a destination on time).

The map portion 5020 includes a representation of a geographical areaincluding objects within a predefined geographical area. In animplementation, the predefined geographical area can include ageographical area corresponding to the geographical area that includesall or at least some portion of the vehicles that are being monitored byone of the vehicle managers. The objects within the geographical areacan include any of the vehicles, and the external objects includingroadways, buildings, and pedestrians. The map portion 5020 can receiveinput including any of touch inputs, voice inputs, and inputs from aninput device. The input to the map portion can generate data about thestate or condition of the selected vehicles or external objects.

In an implementation, the map portion 5020 contains the samerepresentation of objects that are displayed on the fleet managerportion 5010. In another implementation, the number and type of objectsdisplayed between the fleet manager portion 5010 and the map portion5020 can differ. For example, the vehicle manager can zoom into aparticular geographical area thus displaying only a subset of theobjects or vehicles that are represented on the fleet manager portion5010.

The vehicle manager indicator 5030 is a representation of an identifierfor a vehicle manager. Each of the vehicle managers displayed on thefleet manager interface 500 includes a separate vehicle managerindicator. The vehicle manager can be associated with one or morevehicles, which can be distributed or apportioned by the fleet manageror dynamically using machine learning techniques. For example, the fleetmanager can modify the number of vehicles assigned to a vehicle managerincluding any of adding or removing vehicles and transferring one ormore vehicles from one vehicle manager to another vehicle manager.

The vehicle indicator 5040 is a representation of the state or conditionof an autonomous vehicle, the state including any of a vehicle task,vehicle occupancy, vehicle operational mode (e.g. autonomous operationor manual operation), and a vehicle issue, including but not limited toan issue with the operational state of the vehicle. The vehicleindicator 5040 can include various colors, shapes, patterns, text,pictograms, or any combination thereof, to represent aspects of thestate or condition of the vehicle indicator. As an example, the vehicleindicator 5040 can represent an autonomous vehicle that is travelling toa destination in order to pick up a passenger. Further, the vehicleindicator 5040 can represent an autonomous vehicle that is carrying apassenger and travelling to a destination in order to drop the passengeroff.

The vehicle manager assignment queue 5050 is a representation of thevehicles that are assigned to a vehicle manager. The vehicles can beassigned to the vehicle manager assignment queue 5050 by the fleetmanager or by the vehicle managers themselves or automatically usingmachine learning techniques. For example, one vehicle manager mayrealize that they are monitoring too many vehicles and can assign asubset of those vehicles to another vehicle manager that is determinedto have additional monitoring capacity. As shown in FIG. 5 the vehicleindicators (e.g., the vehicle indicator 5040) within the vehicle managerassignment queue 5050 are assigned to the vehicle manager associatedwith the vehicle manager indicator 5030 “Larry.” This vehicle managerindicator can represent the real name of the vehicle manager or ausername or another identifier.

FIG. 6 is a diagram illustrating an example of a vehicle indicator 6000for use in an interface including the fleet manager interface 500illustrated in FIG. 5, a vehicle manager interface 7000 illustrated inFIG. 7 or a vehicle manager interface 9000 illustrated in FIG. 9.

The vehicle indicator 6000 includes a next task indicator 6010, acurrent task indicator 6020, an actual progress indicator 6030, adeviation magnitude indicator 6040, an expected progress indicator 6050,a time scale 6060, a time to completion 6070, an elapsed time 6080, anda time compression indicator 6090.

The next task indicator 6010 can be used to indicate a task that isassigned to the vehicle and that is not currently being performed. Thenext task indicator 6010 can indicate a task that will be performedfollowing performance of a current task (e.g. a current task associatedwith the current task indicator 6020). For example, the next taskindicator 6010 can indicate that a drop off will occur after the currenttask associated with the current task indicator 6020 is completed. In animplementation, an interaction with the next task indicator 6010 (e.g.selecting the next task indicator 6010) can show a description of thetask that will be performed next.

The current task indicator 6020 can be used to indicate a task that iscurrently being performed by a vehicle. For example, the current taskindicator 6020 can include picking up a passenger at a designatedlocation. In an implementation, an interaction with the current taskindicator 6020 (e.g. selecting the current task indicator 6020) can showa description of the task that is currently being performed.

The next task indicator 6010 and the current task indicator 6020 can beassociated with tasks including but not limited to any of, travelling toa pickup destination, including travelling to a destination to pick upone or more passengers or cargo, travelling to a drop-off destination,including travelling to a destination to drop off one or more passengersor cargo, travelling to a maintenance destination, including travellingto a destination at which maintenance or repairs can be performed on thevehicle; and travelling to a refueling destination, including travellingto a destination to refuel the vehicle including petroleum stations orelectrical charging stations.

The shape of the next task indicator 6010 or the current task indicator6020 can correspond to the task that is being performed by the vehicle.For example, the next task indicator 6010, is shown as a circle in FIG.6, which indicates that the vehicle is travelling to a drop offdestination. The circular shape of the current task indicator 6020 canindicate that the vehicle is travelling to a pick up destination. Theshape can include but is not limited to circles, squares, triangles,rectangles, etc.

A pattern (e.g. cross shape, zig-zag) in the next task indicator 6010 orthe current task indicator 6020 can indicate whether the vehicle isoperating in any of an autonomous mode, including a mode in which thevehicle directs itself or the vehicle is being directed remotely by acomputing device, a directed mode, including a mode in which a humanoperator is directing the vehicle from a remote location, a manual mode,including a mode in which a human operator is operating the vehicle fromwithin the vehicle, and a semi-autonomous mode, including a mode inwhich the vehicle can switch between the autonomous mode and the manualmode based on the state of the vehicle and a mode in which the vehicleis simultaneously being controlled using autonomous features and humanoperation. For example, the vehicle mode indicator 4020 is shown as across in FIG. 4, which can indicate, for example, any of theaforementioned modes.

The characteristics, including a fill, of the next task indicator 6010or the current task indicator 6020 can be used to indicate whether thevehicle contains one or more passengers. In an implementation, anoccupied vehicle is indicated by the next task indicator 6010 or thecurrent task indicator 6020 being in a filled state. For example, nofill (e.g. no pattern, no shading, and a light color) could be used toindicate that the vehicle does not contain occupants.

The color of the next task indicator 6010 or the current task indicator6020 can be used to indicate an issue associated with the vehicleincluding any of a passenger issue, including a request for assistancefrom a passenger inside the vehicle, a traffic issue, including issuesrelating to traffic congestion, traffic accidents, and construction, adecision issue, including issues relating to a decision that can be madeby a vehicle manager regarding whether to take control over the vehicle,a physical issue with the state of the vehicle, including issuesrelating to the operational state of the vehicle (e.g. engine state,fuel state). In an implementation, a default color can be used toindicate that the vehicle is operating in a normal state and that noissues with the vehicle are pending.

The actual progress indicator 6030 indicates the actual portion of aroute distance that has been traversed, or time that has passed, on theway to a destination. For example, if the progress indicator is at thehalfway point of the time scale 6060, half of the route distance hasbeen completed, or half of the estimated travel time has passed.

The expected progress indicator 6050 indicates a portion of a routedistance that was estimated to have been completed by the vehicle by thecurrent time and can include a portion of the estimated time to travelto a destination or a portion of the estimated distance that has beentraversed by the vehicle. The deviation magnitude indicator 6040indicates the portion of the route distance or the portion of the traveltime by which the expected progress time (indicated by the expectedprogress indicator 6050) deviates from the actual progress time(indicated by the actual progress indicator 6030).

The time scale 6060 indicates the total travel time or total traveldistance to complete a route for the vehicle. For example, if the timescale 6060 is representative of a total travel time of thirty minutes,half of the time scale 6060 is fifteen minutes. In the event that thetime scale 6060 is for a longer time period, the time compressionindicator 6090 can indicate that a portion of the time scale 6060 thatis not proportional to the remaining part of the time scale 6060 haselapsed. For example, the time compression indicator 6090 can indicatethat half of the time scale 6060 has elapsed. By way of example, theelapsed time 6080 indicates the travel time that has elapsed on the wayto a destination.

Total completion time for a route can be represented by a length of thetime scale 6060 which includes the length of the deviation magnitudeindicator 6040, the length of the time to completion 6070, and thelength of the elapsed time 6080. By way of example, the time tocompletion 6070 indicates the remaining travel time before the vehiclearrives at the destination or completes the associated/assigned task.

FIG. 7 is a diagram illustrating an example of a vehicle managerinterface 7000. The vehicle manager interface 7000 can be generatedbased on one or more instructions that are executable on a computingapparatus, including the controller apparatus 2410 as shown in FIG. 2,and which can be stored in a memory of a computing apparatus, includingthe controller apparatus 2410.

For example, the vehicle manager interface 7000 can be generated by thecontroller apparatus 2410, based on instructions that are interpreted bya client computing device that accesses the controller apparatus 2410through a computer network. The client computing device can thengenerate a representation of the vehicle manager interface 7000 on adisplay device.

The vehicle manager interface 7000 includes a vehicle manager portion7010, a map portion 7020, a vehicle indicator 7030, and a vehicleindicator 7040, any of which can be based on data associated with thestate of physical objects including any of vehicles and external objectsincluding but not limited to pedestrians, cyclists, roadways, andbuildings.

The vehicle manager portion 7010 includes a representation of objectsthat are being monitored or tracked by the vehicle manager utilizing thevehicle manager interface 7000. A plurality of vehicle managers can bemonitoring a plurality of vehicles each with their own specificinterface. The objects can include vehicles, including the vehicle 2100shown in FIG. 2. The objects can be represented as indicators such asthe vehicle indicator 7030, which can be generated as a variety ofimages including but not limited to as a static image, a dynamic image,a moving image, a live photo or video feed, or any combination thereof.Further, the vehicle manager portion 7010 can receive input includingany of touch inputs, voice inputs, and inputs from an input device.

The map portion 7020 includes a representation of a geographical areaincluding objects within the geographical area. The objects within thegeographical area can include any of the vehicles and the externalobjects including roadways, buildings, cyclists, and pedestrians. In animplementation, the map portion 7020 can have similar or differentobjects represented as the objects represented by the vehicle managerportion 7010.

The vehicle indicator 7030 and the vehicle indicator 7040 arerepresentations of the state or condition of an autonomous vehicle,including any of a vehicle task, vehicle occupancy, vehicle operationalmode (e.g. autonomous operation or manual operation), and a vehicleissue, including but not limited to an issue with the operational stateof the vehicle. The vehicle indicator 7030 and the vehicle indicator7040 can include various colors, shapes, patterns, text, or pictograms,to represent aspects of the state or condition of the autonomousvehicle.

As an example, the vehicle indicator 7030 can represent an autonomousvehicle that is travelling to a destination in order to pick up apassenger. Further, the vehicle indicator 7040 can represent anautonomous vehicle that is carrying another passenger and travelling toa destination in order to drop that passenger off. The different tasksor actions that the respective autonomous vehicles are carrying outresult in the graphical display differences between the vehicleindicators 7030 and 7040 (e.g., the vehicle indicator 7030 has a filledin circle and the vehicle indicator 7040 has an unfilled square).

FIG. 8 is a diagram illustrating an example of a vehicle managerinterface 8000. The vehicle manager interface 8000 can be generatedbased on one or more instructions that are executable on a computingapparatus, including the controller apparatus 2410 as shown in FIG. 2,and which can be stored in a memory of a computing apparatus, includingthe controller apparatus 2410.

For example, the vehicle manager interface 8000 can be generated by thecontroller apparatus 2410, based on instructions that are interpreted bya client computing device that accesses the controller apparatus 2410through a computer network. The client computing device can thengenerate a representation of the vehicle manager interface 8000 on adisplay device.

The vehicle manager interface 8000 resembles the vehicle managerinterface 7000 and includes a vehicle manager portion 8010 (similar tothe vehicle manager portion 7010 as shown in FIG. 7), a map portion 8020(similar to the map portion 7020 as shown in FIG. 7), a vehicleindicator 8030, a task control 8040, a resolved control 8050, a callcontrol 8060, and a reroute control 8070, any of which can be based ondata associated with the state of physical objects including but notlimited to at least one of vehicles, roadways, buildings, andpedestrians. In another implementation, the vehicle manager interface8000 includes different control functions other than the task control8040, the resolved control 8050, the call control 8060, and the reroutecontrol 8070 that enable the vehicle manager to interface with andcontrol various aspects of the respective autonomous vehicle or objectbeing monitored or tracked.

The vehicle manager portion 8010 includes a representation of objectsthat are being monitored or tracked. The objects can include vehicles,including the vehicle 2100 shown in FIG. 2. The objects can berepresented as indicators such as the vehicle indicator 8030, which canbe generated as a static image or moving image or a different type ofimage. Further, the vehicle manager portion 8010 can receive inputincluding any of touch inputs, voice inputs, and inputs from an inputdevice. By way of example, vehicle indicators including the vehicleindicator 8030 can be selected by an operator of the vehicle managerinterface 8000, such as a vehicle manager. Selection of the vehicleindicators can generate data about the state or condition of therespective vehicle represented by the vehicle indicators (e.g. theselected vehicle indicator can indicate whether the vehicle will arriveat a destination on time).

The map portion 8020 includes a representation of a geographical areaincluding objects within the geographical area. The objects within thegeographical area can include any of the vehicles and the externalobjects including but not limited to roadways, buildings, cyclists, andpedestrians. The map portion 8020 can receive input including any oftouch inputs, voice inputs, and inputs from an input device. The inputto the map portion can generate data about the state or condition of theselected vehicles or external objects. In an implementation, the mapportion 8020 can have similar or different objects represented as theobjects represented by the vehicle manager portion 8010.

For example, selecting a building, such as a stadium, could generatedata indicating that a sporting event is taking place at the stadiumwithin a certain time frame. Accordingly, the vehicle manager, cananticipate congestion in the vicinity of the stadium at the conclusionof the sporting event, due to the increased traffic flow resulting frompatrons leaving the stadium. Therefore, the vehicle manager can rerouteor change the completion time of one of the autonomous vehicles thatthey are monitoring and that is scheduled to carry out a specific tasknear the stadium at the conclusion of the sporting event.

The vehicle indicator 8030 includes a representation of the state orcondition of a vehicle (e.g. an autonomous vehicle or a human drivenvehicle) and includes any of a vehicle task, vehicle occupancy, vehicleoperational mode (e.g. autonomous operation or manual operation), and avehicle issue, including but not limited to an issue with theoperational state of the vehicle. The vehicle indicator 8030 can includevarious characteristics including colors, shapes, patterns, text, orpictograms, to represent aspects of the state or condition of thevehicle indicator. As an example, the vehicle indicator 8030 canrepresent an autonomous vehicle that is travelling to a destination inorder to pick up a passenger. Or the vehicle indicator 8030 canrepresent an autonomous vehicle that is carrying a passenger andtravelling to a destination in order to drop the passenger off.

Any of the task control 8040, the resolved control 8050, the callcontrol 8060, and the reroute control 8070, can be controlled ormodified based on an input including any of a user input based on aninput received through an input device including a tactile input device(e.g. a keyboard, mouse, or touchscreen), an audio input device (e.g. amicrophone), and a visual input device (e.g. a camera). Moreover, any ofthe task control 8040, the task resolved control 8050, the call control8060, and the reroute control 8070, can be controlled or modified basedon instructions, such as computer program instructions (e.g.instructions to select vehicle indicators that meet pre-establishedcriteria such as a common destination).

The task control 8040 can be used to modify the task that is associatedwith a vehicle. For example, the vehicle associated with the vehicleindicator 8030 may have completed a drop off. A vehicle manager caninteract with the task control 8040 and modify the vehicle task toindicate that the vehicle should now pick up a passenger instead ofcompleting a previously assigned next task. The task can be modifiedand/or updated while the task completion is in progress or in regards toan upcoming task. For example, the current task can be set for deliveryof a package at a certain time but based on traffic conditions, thecurrent task is updated to pick up a passenger nearby and drop them offat a location that isn't within the traffic congestion area. In anotherexample, one of the upcoming tasks of the vehicle can bemodified/updated/deleted while the vehicle is completing a currentunrelated task.

The resolved control 8050 can be used to indicate that an issued relatedto the vehicle has been resolved or completed by the vehicle manager.For example, after a vehicle manager receives a help request from avehicle associated with the vehicle indicator 8030 or a passenger of thevehicle, and provides assistance to the vehicle, the resolved control8050 can be activated by the vehicle manager to indicate that the issuehas been resolved and is no longer pending. In an implementation,activating the resolved control 8050 can modify vehicle data associatedwith the vehicle including a vehicle task urgency that includes anindication of the urgency of a vehicle request or a vehicle task (e.g.an ambulance carrying a patient to a hospital). For example, a vehiclecarrying a patient in urgent need of medical help, could send a requestto a vehicle manager for an optimized rerouting and once the vehiclemanager takes care of this request or concludes that additional help isneeded, the vehicle manager can interact with the resolved control 8050to update the status of the request.

The call control 8060 can used to contact and communicate with thevehicle associated with the vehicle indicator 8030. For example, whenthe call control 8060 is activated, a vehicle manager can interact withan occupant or passenger of the vehicle associated with the vehicleindicator 8030. In an implementation, when the call control 8060 isactivated, any of an audio connection or an audio and video connection(e.g., live video communication feeds) can be established with thevehicle associated with the vehicle indicator 8030.

The reroute control 8070 can be used to modify a route associated with avehicle. For example, the vehicle associated with the vehicle indicator8030 could be in transit to a destination via a route that will passthrough heavily congested traffic. The reroute control 8070 could beused to reroute the vehicle to avoid entering the area with the heavilycongested traffic. In another implementation, the reroute control 8070can be a different type of control that provides tele-operation ofautonomous vehicles

FIG. 9 is a diagram illustrating an example of a vehicle managerinterface 9000. The vehicle manager interface 9000 can be generatedbased on one or more instructions that are executable on a computingapparatus, including the controller apparatus 2410, and which can bestored in a memory of a computing apparatus, including the controllerapparatus 2410.

For example, the vehicle manager interface 9000 can be generated by thecontroller apparatus 2410, based on instructions that are interpreted bya client computing device that accesses the controller apparatus 2410through a computer network. The client computing device can thengenerate a representation of the vehicle manager interface 9000 on adisplay device.

The vehicle manager interface 9000 resembles the vehicle managerinterface 8000 as shown in FIG. 8 and includes a vehicle manager portion9010, a map portion 9020, a vehicle indicator 9030, a vehicle indicator9040, a cluster control 9050, and an area selection control 9060, any ofwhich can be based on data associated with the state of physical objectsincluding but not limited to at least one of vehicles, roadways,buildings, and pedestrians.

The vehicle manager portion 9010 includes a representation of objectsthat are being monitored or tracked. The objects can include vehicles,including the vehicle 2100 shown in FIG. 2. The objects can berepresented as indicators such as the vehicle indicator 9030, which canbe generated as a static image or moving image or any other type ofimage. Further, the vehicle manager portion 9010 can receive inputincluding any of touch inputs, voice inputs, and inputs from an inputdevice.

By way of example, vehicle indicators including the vehicle indicator9030 and the vehicle indicator 9040 can be selected by an operator, suchas a vehicle manager. Selection of the vehicle indicators can generatedata about the state or condition of the respective vehicle representedby the vehicle indicators (e.g. the selected vehicle indicator canindicate whether the vehicle will arrive at a destination on time).

The map portion 9020 includes a representation of a geographical areaincluding objects within the geographical area. The objects within thegeographical area can include any of the vehicles, and the externalobjects including but not limited to roadways, buildings, cyclists andpedestrians. The map portion 9020 can receive input including any oftouch inputs, voice inputs, and inputs from an input device. The inputto the map portion can generate data about the state or condition of theselected vehicles or external objects. For example, selecting abuilding, such as a stadium, could generate data indicating that asporting event is taking place at the stadium within a certain timeframe. Accordingly, the vehicle manager, can anticipate congestion inthe vicinity of the stadium at the conclusion of the sporting event, dueto the increased traffic flow resulting from patrons leaving the stadiumand reroute the vehicle.

The vehicle indicator 9030 and the vehicle indicator 9040 arerepresentations of the state or condition of two separate autonomousvehicles, including any of a vehicle task, vehicle occupancy, vehicleoperational mode (e.g. autonomous operation or manual operation), and avehicle issue, including an issue with the operational state of thevehicle. The vehicle indicator 9030 and the vehicle indicator 9040 caninclude various colors, shapes, patterns, text, or pictograms, torepresent aspects of the state or condition of the vehicle indicator. Asan example, the vehicle indicator 9030 can represent an autonomousvehicle that is travelling to a destination in order to pick up apassenger. Further, the vehicle indicator 9040 can represent anautonomous vehicle that is carrying another passenger and travelling toa destination in order to drop that passenger off.

The cluster control 9050 and the area selection control 9060 include acontrol element that can be controlled or modified based on an inputincluding any of a user input based on an input received through aninput device including a tactile input device (e.g. a keyboard, mouse,or touchscreen), an audio input device (e.g. a microphone), and a visualinput device (e.g. a camera). Moreover, the cluster control 9050 and thearea selection control 9060 element can be controlled or modified basedon instructions, such as computer program instructions (e.g.instructions to select vehicle indicators that meet pre-establishedcriteria such as a common destination).

The area selection control 9060 is a control element that can becontrolled or modified to select a section of the map portion 9020. Forexample, a rectangular section of the map can be selected or highlightedby the vehicle manager in order to define that the vehicles within thegeographical area of the map will be monitored and selected or clusteredon the vehicle manager portion 9010 that corresponds to the selectedpart of the map portion 9020. The objects within the selected section ofthe map portion 9020 can be monitored and organized according to acluster criteria including any of routes, destinations, and points ofdeparture in common. As illustrated in FIG. 9, the cluster control 9050can indicate that the vehicle indication 9030 and vehicle indication9040 are part of a cluster group based on shared cluster criteria (e.g.,they share similar routes).

FIG. 10 is a diagram illustrating an example of a vehicle managerinterface 10000. The vehicle manager interface 10000 can be generatedbased on one or more instructions that are executable on a computingapparatus, including the controller apparatus 2410 as shown in FIG. 2,and which can be stored in a memory of a computing apparatus, includingthe controller apparatus 2410.

For example, the vehicle manager interface 10000 can be generated by thecontroller apparatus 2410, based on instructions that are interpreted bya client computing device that accesses the controller apparatus 2410through a computer network. The client computing device can thengenerate a representation of the vehicle manager interface 10000 on adisplay device.

The vehicle manager interface 10000 includes a vehicle indicator 10010,a path indicator 10020, an external object indicator 10030, anobstruction indicator 10040, a pedestrian indicator 10050, any of whichcan be based on data associated with the state of physical objectsincluding but not limited to at least one of vehicles, roadways,buildings, and pedestrians. A plurality of configurations of externalobjects, obstructions, pedestrians, and any combination thereof can bedisplayed on the vehicle manager interface 10000. The vehicle indicator10010 can be used to represent a vehicle. In this example, the vehicleis represented as a three dimensional model, however the vehicleindicator 10010 can be represented in different ways including any of atwo dimensional image and a pictogram, such as an icon.

The path indicator 10020 can be used to represent a path between thecurrent vehicle location and a vehicle destination. In animplementation, a vehicle manager can guide the vehicle associated withthe vehicle indicator 10010 along the path indicated by the pathindicator 10020. For example, when providing remote assistance to avehicle associated with the vehicle indicator 10010, a path indicatorsuch as a virtual lane can be generated in order to provide a visualrepresentation of the path that the vehicle can travel on that isillustrated by the path indicator 10020.

The external object indicator 10030 can be used to represent externalobjects such as other vehicles that could change the intended route ofthe vehicle as an example. The obstruction indicator 10040 can be usedto represent external objects that can obstruct the movement of thevehicle represented by the vehicle indicator 10010. The pedestrianindicator 10050 can be used to represent an external object including apedestrian or a cyclist or another moving object. The pedestrianindicator 10050 can be indicated with a distinctive color scheme that isdifferent from other external objects represented by the external objectindicator 10030 or the obstruction indicator 10040. In this way,pedestrians can be distinguished from other types of external objects toprovide additional awareness and avoidance capabilities. In animplementation, the external object indicator 10030, the obstructionindicator 10040, and the pedestrian indicator 10050, or any combinationthereof, can be represented by the same or similar type of indicatorthat covers all objects that could affect at least one parameter (e.g.,route, travel time, etc.) of the vehicle represented by the vehicleindicator 10010.

The steps, or operations, of any method, process, or algorithm describedin connection with the implementations of the disclosed technologyherein, may be implemented in hardware, firmware, software executed byhardware, circuitry, or any combination thereof. To facilitateexplanation, the processes 11000-13000, shown in FIGS. 11-13, aredepicted and described as a series of operations. However, theoperations in accordance with this disclosure can occur in variousorders or concurrently. Additionally, operations in accordance with thisdisclosure may occur with other operations not presented and describedherein.

FIG. 11 is a flow chart of a technique 11000 for providing remotesupport of autonomous operation of vehicles in accordance with thepresent disclosure. In an implementation, the technique 11000 isutilized by a vehicle monitoring system or remote support system thatincludes any of a fleet manager, vehicle manager, and the aforementionedinterfaces. Some or all aspects of the technique 11000 for vehicleprocessing may be implemented in a vehicle including the vehicle 1000shown in FIG. 1, the vehicle 2100 shown in FIG. 2, or a computingapparatus including the controller apparatus 2410 shown in FIG. 2. In animplementation, some or all aspects of the technique 11000 for vehicleprocessing can be implemented in a system combining some or all of thefeatures described in this disclosure.

At operation 11010, status or state data is received from one or more ofthe vehicles. In an implementation, the state data is received by acommunication system or similar device of the vehicle monitoring system.The one or more vehicles can include a device or apparatus (e.g. aconveyance) that is used to transport objects including any of apassenger and cargo. The one or more vehicles can include an autonomousvehicle or a vehicle that is driven by a human driver or asemi-autonomous vehicle.

The state data includes but is not limited to data based on the state orcondition of the vehicles, including any of kinetic data relating to anyof the velocity and acceleration of a vehicle, location data, includingthe geographical location of a vehicle (e.g. the latitude and longitudeof the vehicle) or the location of the vehicle with respect to anotherobject, vehicle position, including the orientation and inclination(e.g. slope of the vehicle on an incline) of the vehicle, theoperational state of the vehicle, including the electrical state ormechanical state of the vehicle (e.g. health of the electrical vehiclesystems, mechanical vehicle systems, tire pressure, etc.), maintenancedata related ongoing maintenance of the vehicle, energy source dataincluding an amount of fuel remaining or an amount of battery chargeremaining, sensor data based on outputs from sensors including, opticalsensors, audio sensors, an motion sensors, internal state data,including a temperature and humidity inside the passenger cabin of thevehicle, and a current task (e.g., pick up a passenger) of the vehicle.

In an implementation, the state data can include a request forassistance from a vehicle, including a first vehicle. The first vehiclecan include any of a vehicle that is selected from among the vehiclesand a vehicle that is ranked or prioritized as first or high prioritybased on the state data associated with the vehicle. In animplementation, the sensor data can include image data received from avehicle including the first vehicle. The image data can comprise acamera image including any of an image of an environment external to thefirst vehicle and an image of an occupant within or the interior of thefirst vehicle.

At operation 11020, a map is generated. In an implementation, a firstmap display of a defined geographical area is generated. The definedgeographical area can be predetermined by an operator or automaticallydefined based on historical data or machine learning techniques. Thefirst map display includes at least some or all of the vehicles withinthe defined geographical area. The vehicles in the defined geographicalarea can be represented as vehicle map indicators. In an implementation,in response to an input to a state indicator, an appearance of a vehiclemap indicator representing a respective vehicle within the first mapdisplay is modified.

In an implementation, a second map display of another definedgeographical area or the same defined geographical area that wasutilized for the first map display is generated. The second map displayincludes the vehicles within the another defined geographical area andassigned to a first remote support queue as vehicle map indicators. Thesecond map display can be displayed concurrently with another display,including any of the first state display and a second state display. Inan implementation, the second map display can be generated as asemi-transparent overlay that is superimposed on top of another displayincluding the first map display.

At operation 11030, a first status or state display is generated. Thefirst state display includes state indicators corresponding to the statedata of at least some of the vehicles that are associated with the firststate display. The state indicators can include any of glyphs, images,text, shapes, pictograms, and any combination thereof. The first statedisplay can be concurrently displayed with the first map display.

The state indicators can indicate any of an operating mode of thevehicle indicating whether the vehicle is receiving remote support orhas requested remote support, an expected progress in completing thecurrent task, an actual progress in completing the current task, aprogress indicator indicating a deviation (if any) between the actualprogress and the expected progress, the progress indicator indicatingwhether the vehicle is any of early, late, and on-time, and an issuerelated to operation of the vehicle. The state indicators indicate theaforementioned information using any of a color, a shape, and a size ofat least a portion of a state indicator representing the vehicle in astate display including any of the first state display and the secondstate display, a timescale with a length based on an expected amount oftime to complete the current task, the timescale including a linerepresenting the actual progress and the progress indicator extendingfrom the line, and along a length of the timescale indicating thedeviation, a color of the progress indicator indicating whether thevehicle is any of early, late, and on time.

In an implementation, in response to an input to a vehicle mapindicator, an appearance of the state indicator representing therespective vehicle is modified within the first state display. An inputsuch as an input from an input device can cause a change in the stateindicator including any of a change in color, size, shape, brightness,and text. For example, the text on the state indicator can be changed orupdated from “pick up” to “drop off” when a passenger has been droppedoff.

In an implementation, the vehicles are assigned to respective remotesupport queues, and the corresponding state indicators for the vehiclesare generated for display in lanes corresponding to the respectiveremote support queues, for example, as shown in FIG. 5. For example, ifthere are ten vehicles being monitored overall, and there are five lanescorresponding to remote support queues that are each managed by aseparate vehicle manager, an operator can assign two vehicles and theircorresponding state indicators to each of the five lanes.

In an implementation, the state indicators can be moved between theremote support queues. For example, the state indicator for a vehiclecan be moved by an operator (such as a fleet manager of the vehiclemonitoring system) from a first lane associated with a first remotesupport queue to a second lane associated with a second remote supportqueue. By moving or reassigning the state indicators of thecorresponding vehicles between remote support queues, the system canensure that the vehicles are monitored and remotely supported properly.As another example, a fleet manager can assign a group of vehicles froma first vehicle manager to a second vehicle manager by changing thelocation of state indicators from a first remote support queuecorresponding to the first vehicle manager to a second remote supportqueue corresponding to the second vehicle manager.

In an implementation, a second state display for a first remote supportqueue is generated. The second state display includes a representationof the state data of the vehicles assigned to the first remote supportqueue as respective state indicators arranged based on a level ofurgency or priority associated with the vehicle represented by the stateindicator. For example, a vehicle that is an ambulance on the way todrop off a passenger at a hospital could be associated with a high levelof urgency. The level of urgency can be based on any of the state dataassociated with the vehicle, preferences, historical data, and real-timetraffic data. Based on the level of urgency, the position or display ofthe state indicators can be modified. For example, a higher level ofurgency (or higher urgency level) can correspond to a higher position(e.g., along a vertical axis) or can correspond to a larger stateindicator in comparison to smaller state indicators that representvehicles with lower levels of urgency.

In an implementation, the state indicators of the first state displaycomprise the state indicators of the second state display and elementsof the state indicators of the first state display comprise a subset ofelements of the state indicators of the second state display. Forexample, to facilitate grouping of large numbers of state indicators,the state indicators in the first state display can be simplified toshow fewer details with respect to the state data of a vehicle. Inanother implementation, elements of the state indicators of the secondstate display comprise a subset of elements of the state indicators ofthe first state display.

At operation 11040, a remote support interface is generated. The remotesupport interface can be responsive to an input signal to any stateindicators of the vehicles including but not limited to an input signalto a first state indicator of a first vehicle from the first statedisplay. For example, an interaction with the first state indicator cangenerate a remote support interface with a task control, resolvedcontrol, reroute control, and call control, such as in the interfaceshown in FIG. 8. The remote support interface can be superimposed on thefirst state display and in an area surrounding the state indicator thathas been interacted with using the input signal or the remote supportinterface can be a new display that is generated in response to theinput signal.

In an implementation, a portion of the remote support interface isallocated to display any of an indication of sensor data including thecamera image and an indication of map data representing the camera imagein which the camera image comprises the image of the environmentexternal of the first vehicle.

At operation 11050, instruction data is transmitted to any of thevehicles, such as a first vehicle. The instruction data includesinstructions to the vehicle (e.g. the autonomous vehicle) forperformance by the vehicle or by an occupant of the vehicle (e.g. adriver). By way of example, the instruction data includes but is notlimited to any of controlling movement of a vehicle, including changingthe acceleration, velocity, or direction (e.g. steering), of thevehicle, turning some or all parts of the vehicle on or off (e.g.,turning on a light external to the vehicle), activating or deactivatinga control system in the vehicle including mechanical control systems andelectrical control systems, activating or deactivating sensors in thevehicle (e.g. activating a camera to view inside the vehicle or outsidethe vehicle), activating or deactivating a communication system thatprovides communications with the vehicle, occupants of the vehicle, andobjects or individuals external to the vehicle.

In an implementation, transmitting the instruction data to the firstvehicle can be initiated in response to any of receiving a signal totransmit the instruction data, including a command issued by apre-programmed script, and using the remote support interface (e.g. auser interaction with the remote support interface). The instructiondata can also be transmitted to a vehicle automatically in response to adetected state or situation based on the state data of the vehicle.

In an implementation, the instruction data includes a virtual lanewithin the map data to the first vehicle for the autonomous operation ofthe first vehicle to use for navigation of the first vehicle. Forexample, the instruction data can generate an image that can be includedin a representation or model of an area within a predetermined distanceof the vehicle, for example, the path indicator 10020 shown in FIG. 10.The virtual lane can help guide the autonomous vehicle around variousobstructions or external objects that prompted the requirement ofreceiving the instruction data.

At operation, 11060, the vehicles are ranked according to a level ofurgency for remote support. In an implementation, the vehicles in thefirst state display or the second state display or both are rankedaccording to the level of urgency. The level of urgency for the remotesupport can be determined based on the state data or user inputs orother types of aggregated data (e.g., time of day, typical urgencylevels for similar vehicles garnered by analyzing historical data,etc.). For example, state data that indicates that a vehicle is thirtyminutes behind schedule could have a greater level of urgency than avehicle that is only twenty seconds behind schedule.

At operation 11070, the state indicators are arranged in the first statedisplay based on the corresponding level of urgency. In animplementation, the arrangement can cluster similar urgency levels orlist them from top to bottom in ascending or descending order. Forexample, the state indicators representing at least some of the vehiclescan be moved from a lower portion of the first state display to an upperportion of the first state display in accordance with an increasinglevel of urgency. In another example, state indicators associated withlower levels of urgency can be displayed on the leftmost portion of adisplay with the state indicators being moved to the right as the levelof urgency increases. In addition, state indicators that are below acertain urgency threshold level can be hidden from the display.

FIG. 12 is a flow chart of a technique 12000 for providing remotesupport of autonomous operation of vehicles in accordance with thepresent disclosure. Some or all of the technique 12000 for vehicleprocessing may be implemented in a vehicle including the vehicle 1000shown in FIG. 1, the vehicle 2100 shown in FIG. 2, or a computingapparatus including the controller apparatus 2410 shown in FIG. 2. In animplementation, some or all aspects of the technique 12000 for vehicleprocessing can be implemented in a system combining some or all of thefeatures described in this disclosure.

At operation 12010, status or state data is received from one or more ofthe vehicles. The state data includes an indication of the state orcondition of the vehicles, such as, for example, the state datadescribed in the process 11000. At operation 12020, first level controlstations are generated. The first level control stations can includeinstances that can receive inputs, process data including the statedata, and produce an output including a display output. The first levelcontrol stations include respective first state displays that displaysstate indicators of the state data from a portion of the vehiclesassigned to a respective one of the first level control stations. Thefirst level control stations can comprise remote support queues andcorresponding lanes and can be managed by an operator such as a vehiclemanager.

At operation 12030, a second status or state display is generated for asecond level control station. In an implementation, the second levelcontrol station is a fleet manager that manages the first level controlstations and corresponding vehicle managers. In another implementation,multiple second state displays are generated for respective second levelcontrol stations. The second state display displays the state data ofthe vehicles and can either display similar or different state data thatis displayed on the first state display. For example, the second levelcontrol station can correspond to a control station operated by a fleetmanager and the first level control station can correspond to a controlstation operated by a vehicle manager under the authority of the fleetmanager

At operation 12040, in response to receiving an indication that avehicle is requesting support, instruction data is transmitted to thevehicle. In an implementation, the instruction data is transmittedwithout receiving the indication (e.g., a request for support). Theinstruction data can be transmitted to a first vehicle displayed in oneof the first state displays of the first level control stations. Theinstruction data can include instructions for remote operation of thevehicle, including the instruction data described in the technique 11000in FIG. 11.

At operation 12050, a workload is allocated or balanced or optimizedbetween the first level control stations by an operator (such as a fleetmanager) that assigns the vehicles using the state indicators of thesecond state display. In an implementation, the assignment is automatedbased upon a detection by the system that there is an imbalance betweenthe workload or one of the vehicle managers is in urgent need of help(e.g., the vehicle manager is monitoring too many vehicles or one of thevehicles requires time-consuming support and thus the other vehiclesshould be reassigned for a predetermined time period). The workload canbe based on any of the state data, external data, historical data, andthe distribution of the vehicles to the first level control stations.

FIG. 13 is a flow chart of a technique 13000 for providing remotesupport of autonomous operation of vehicles in accordance with thepresent disclosure. Some or all aspects of the technique 13000 forvehicle processing may be implemented in a vehicle including the vehicle1000 shown in FIG. 1, the vehicle 2100 shown in FIG. 2, or a computingapparatus including the controller apparatus 2410 shown in FIG. 2. In animplementation, some or all aspects of the technique 13000 for vehicleprocessing can be implemented in a system combining some or all of thefeatures described in this disclosure.

At operation 13010, a map display of a defined geographical area isgenerated. In an implementation, the map display can include a first mapdisplay of a first defined geographical area that is generated forrespective first level control stations. The first map display includesvehicle map indicators to indicate the vehicles within the definedgeographical area and assigned to the respective one of the first levelcontrol stations as vehicle map indicators. The first map display can beconcurrently displayed with the respective first state display.

At operation 13020, another map display of a defined geographical areais generated. In an implementation, the another map display is a secondmap display of the first defined geographical area (or another definedgeographical area) that is generated for a second control station. Thesecond map display includes vehicle map indicators to indicate thevehicles within the first defined geographical area and assigned to thesecond control station as the vehicle map indicators. The second mapdisplay can be concurrently displayed with the respective second statedisplay. At operation 13030, an input selecting a portion of a first mapdisplay of a respective first level control station is received. Forexample, the input can include a selection signal from an input device.

At operation 13040, the state indicators of the first state display areclustered corresponding to the selection of operation 13030. In animplementation, the state indicators of the first state display of therespective first level control station are clustered based on respectivelocations of those of the vehicles assigned to the first level controlstation relative to a location of the selected portion of the first mapdisplay. For example, the vehicles that are within a predetermineddistance of each other can be clustered together by selecting a portionof the first map display that encompasses or surrounds the vehicles thatare within a predetermined distance of each other. In an implementation,the clustering of the state indicators can be based on the state data orobserved data or preferences.

At operation 13050, instruction data is transmitted to all or a portionof the vehicles within a cluster. In an implementation, the instructiondata is transmitted to the second vehicle based on the instruction datathat has been sent or transmitted to the first vehicle. The instructiondata can include instructions to control or communicate with the vehicleand can include the instruction data described in the technique 11000 ofFIG. 11. In an implementation, the transmission of the instruction datacan be initiated based on an input to a second state indicatorrepresenting a second vehicle in a cluster in common with the firststate indicator representing the first vehicle.

FIG. 14 illustrates a method 14000 for remote support in accordance withthe present disclosure. The method 14000 includes receiving state datafrom the vehicles, via 14010, generating, for first level controlstations, a respective first state display that displays the state datafrom a portion of the vehicles assigned to a respective one of the firstlevel control stations as respective state indicators, via 14020,generating, for a second level control station, a second state displaythat displays the state data of the vehicles, via 14030, generating aremote support interface including the first state display and imagedata received from a first vehicle of the vehicles, via 14040,transmitting, using a state indicator representing a first vehicle inone of the first state displays, instruction data to the first vehiclebased on an indication that the first vehicle needs remote support, theinstruction data, once received by the first vehicle, modifyingautonomous operation of the first vehicle, via 14050, and allocating aworkload between the first level control stations by assigning thevehicles using the state indicators of the second state display, via14060.

The disclosed technology offers the benefits of a way to moreeffectively organize data relating to the operation of autonomousvehicles in a transportation network. The disclosed technology canreceive state data indicative of the state of an autonomous vehicle,which can be used to prioritize the vehicles in order to moreefficiently route vehicles that are in transit and assist vehicles aremalfunctioning or adversely affected by an external factor. Further, thedisclosed technology provides an improved way for human operators toremotely control autonomous vehicles, thereby reducing disruptions inthe transportation network.

While the disclosed technology has been described in connection withcertain embodiments, it is to be understood that the disclosedtechnology is not to be limited to the disclosed embodiments but, on thecontrary, is intended to cover various modifications and equivalentarrangements included within the scope of the appended claims, whichscope is to be accorded the broadest interpretation so as to encompassall such modifications and equivalent structures as is permitted underthe law.

What is claimed is:
 1. A remote support apparatus comprising: a memory;and a processor configured to execute instructions stored in the memoryto: receive state data from vehicles; generate a first map displayincluding a representation of a geographical area and vehicle mapindicators representing a portion of the vehicles within thegeographical area; generate a first state display including arepresentation of the portion of the vehicles as state indicators;generate a remote support interface including the first state displayand image data received from a first vehicle of the portion of thevehicles, wherein the remote support interface is responsive to an inputsignal to a first state indicator of the first vehicle from the firststate display; transmit instruction data to the first vehicle to modifyan autonomous operation of the first vehicle based on the input signalto the remote support interface; rank the vehicles according to a levelof urgency based on the state data; position the state indicators in thefirst state display based on the level of urgency corresponding to thestate indicators; assign the vehicles to respective remote supportqueues; generate the first state display by displaying lanes, each lanecorresponding to a respective remote support queue, the state indicatorsarranged in the lanes based on an assignment of at least one vehiclecorresponding to the respective remote support queue; and assign avehicle from a first one of the remote support queues to a second one ofthe remote support queues by moving a respective state indicator from afirst lane representing the first one of the remote support queues to asecond lane representing the second one of the remote support queues. 2.The apparatus of claim 1, wherein the first state display is displayedconcurrently with the first map display.
 3. The apparatus of claim 1,wherein the processor is further configured to execute instructionsstored in the memory to: receive a request for assistance from the firstvehicle; and responsive to the request for assistance, change anappearance of the first state indicator in the first state display. 4.The apparatus of claim 1, wherein the image data comprises a cameraimage including any of an image of an environment external to the firstvehicle and an image of an occupant within the first vehicle, whereinthe remote support interface displays the camera image.
 5. The apparatusof claim 4, wherein the processor is further configured to executeinstructions stored in the memory to: generate the remote supportinterface by including map data representing the image of theenvironment external to the first vehicle; and transmit the instructiondata to the first vehicle to modify the autonomous operation of thefirst vehicle by transmitting a virtual lane within the map data to thefirst vehicle for use in navigation of the first vehicle.
 6. Theapparatus of claim 1, wherein the processor is further configured toexecute instructions stored in the memory to: display the state data ofa vehicle by displaying any of a current task of the vehicle, anoperating mode of the vehicle that indicates whether the vehicle isreceiving remote support, a deviation of the vehicle from an expectedprogress of the current task, whether the deviation is ahead of orbehind the expected progress, and an issue related to operation of thevehicle, wherein the display utilizes at least one of a color, a shape,and a size of at least a portion of a state indicator representing thevehicle in the first state display.
 7. The apparatus of claim 6, whereinthe processor is further configured to execute instructions stored inthe memory to: generate a first map display of a defined geographicalarea that displays those of the vehicles within the defined geographicalarea as vehicle map indicators, the first map display displayedconcurrently with the first state display.
 8. The apparatus of claim 7,wherein the processor is further configured to execute instructionsstored in the memory to: in response to an input to a state indicatorassociated with a respective vehicle, modify an appearance of a vehiclemap indicator representing the respective vehicle within the first mapdisplay; and in response to an input to the vehicle map indicator,modify an appearance of the state indicator representing the respectivevehicle within the first state display.
 9. The apparatus of claim 7,wherein the vehicles are assigned to respective remote support queues,and the instructions include instructions to: generate a second statedisplay for a remote support queue that displays the state data of thoseof the vehicles assigned to the remote support queue as respective stateindicators arranged based on the level of urgency; and generate a secondmap display of a second defined geographical area that displays those ofthe vehicles within the second defined geographical area and assigned tothe remote support queue as vehicle map indicators, the second mapdisplay displayed concurrently with the second state display.
 10. Theapparatus of claim 9, wherein the state indicators of the first statedisplay comprise the state indicators of the second state display andelements of the state indicators of the first state display comprise asubset of elements of the state indicators of the second state display.11. The apparatus of claim 10, wherein the state data comprises any of acurrent task of the vehicle, an operating mode of the vehicle indicatingwhether the vehicle is receiving remote support, an expected progress incompleting the current task, an actual progress in completing thecurrent task, a progress indicator indicating a deviation between theactual progress and the expected progress, a progress indicatorindicating whether the vehicle is early, late, or on time, and an issuerelated to operation of the vehicle.
 12. The apparatus of claim 11,wherein the state indicator representing the vehicle in the second statedisplay includes a timescale with a length based on an expected amountof time to complete the current task, the timescale including a linerepresenting actual progress and a progress indicator extending from theline and along a length of the timescale indicating a deviation, a colorof the progress indicator indicating whether the vehicle is any ofearly, late, and on-time.
 13. A method for remote support of autonomousoperation of vehicles, the method comprising: receiving state data fromthe vehicles; generating a first map display of a first definedgeographical area that displays those of the vehicles within the firstdefined geographical area as vehicle map indicators; generating a firststate display that displays the state data from a portion of thevehicles as state indicators; generating a remote support interfaceincluding the first state display and image data received from a firstvehicle of the vehicles, wherein the remote support interface isresponsive to an input signal to a first state indicator of the firstvehicle from the first state display; transmitting, using the remotesupport interface, instruction data to the first vehicle based on anindication that the first vehicle needs remote support, the instructiondata, once received by the first vehicle, modifying autonomous operationof the first vehicle ranking the vehicles according to a level ofurgency based on the state data; positioning the state indicators in thefirst state display based on the level of urgency corresponding to thestate indicators; assigning the vehicles to respective remote supportqueues; generating the first state display by displaying lanes, eachlane corresponding to a respective remote support queue, the stateindicators arranged in the lanes based on an assignment of at least onevehicle corresponding to the respective remote support queue; andassigning a vehicle from a first one of the remote support queues to asecond one of the remote support queues by moving a respective stateindicator from a first lane representing the first one of the remotesupport queues to a second lane representing the second one of theremote support queues.
 14. The method of claim 13, further comprising:receiving a selection of a portion of a first map display of a firstlevel control station; clustering the state indicators of the firststate display of the first level control station based on respectivelocations of those of the vehicles assigned to the first level controlstation relative to a location of the portion of the first map display;and transmitting, using a second state indicator representing a secondvehicle in a cluster in common with the state indicator representing thefirst vehicle, instruction data to the second vehicle based on theinstruction data transmitted to the first vehicle.